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Aligning  with  Mission  Requirements 


Consulting 


Developing  Mission-focused  LREC 
Capability 


This  model  was  originally  presented  at  the  2013  United  States  Special  Operations  Command  (USSOCOM)  Command  Language  Program 
Manager  (CLPM)  Advanced  Competencies  Course  in  a  presentation  titled.  The  Language  Needs  Assessment  Process  and  Resulting  Products. 
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Need  to  take  a  systematic  approach 
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•  Planning,  developing  and  implementing  an 
assessment  in  immersion  or  iso-immersion 
context  is  no  different  than  others— you  need 
objectives  and  a  clear  plan  or  blueprint  for 
success 

—  Have  observed  immersion  exercises  with  no  clear 
assessment  of  learning  or  assessment  objectives 

-  Systematic  approach  is  needed  to  ensure  utility 
and  ROI  of  immersion 

•  What  is  the  purpose  of  assessment? 
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Does  Immersion  Fit  Strategy? 
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Develop  Assessment  Strategy 


TF  ^ 

What  and  when  to  assess? 


How  to  assess  it? 


•  Purpose  of  assessment- 
formative  or  summative 

•  Must  match  with  Needs, 
Objectives  and  Learning 
Strategy  {what,  when,  how 
of  learning) 

•  What  is  important  for  the 
learner  to  demonstrate  at 
this  point? 


•  Must  be  appropriate  for  the 

purpose  and  objectives 

•  Does  method  allow  learner 
to  demonstrate  status  on 
objectives? 

•  support  desired  inference? 

•  Immersion  allows 
demonstration  of  behavior 
(performance)  in  Context 
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Demonstrating  Behavior  (Performance) 
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*  Role-play,  Videogame, 
Simulation,  Assessment 
Center,  Work  Sample, 
Field  Exercise  and 
Immersion  methods 
allow  measurement  of 
behavior  in  CONTEXT 

to  various  degrees  of 

fidelity 

•  Techniques  can  be  very 
similar 


•  Physical  v.  psychological 
fidelity 

•  Johns  (2006)  Discrete 
Context— task,  social 
and  physical 

•  4Ps:  Purpose, 
Practicality,  Parsimony 
and  Price 

•  Best  practices  from 
other  areas  can  inform 
immersion  assessment 
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From  International  Task  Force  on  Assessment  Center  Guidelines: 

1.  Job  analysis/competency  modeling  -  A  job  analysis  of  relevant  behaviors  must 
be  conducted  to  determine  the  dimensions  or  competencies  important  to  job 
success  in  order  to  identify  what  should  be  evaluated  by  the  assessment  center. 

2.  Behavioral  classification  -  Behaviors  displayed  by  participants  must  be  classified 
into  meaningful  and  relevant  categories  such  as  behavioral  dimensions, 
attributes,  characteristics,  aptitudes,  qualities,  skills,  abilities,  competencies,  or 
knowledge. 

3.  Assessment  techniques  -  The  techniques  used  in  the  assessment  center  must  be 
designed  to  provide  information  for  evaluating  the  dimensions  previously 
determined  by  the  job  analysis. 

4.  Multiple  assessments  -  Multiple  assessment  techniques  [activities]  must  be 
used.  The  assessment  techniques  are  developed  or  chosen  to  elicit  a  variety  of 
behaviors  and  information  relevant  to  the  selected  dimensions. 

5.  Simulations  -  The  assessment  techniques  must  include  a  sufficient  number  of 
job-related  simulations  to  allow  opportunities  to  observe  the  candidate's 
behavior  related  to  each  dimension/competency  being  assessed. 
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Naval  Special  Warfare's 
ISO  Immersion 


Tactical  Iraqi  Role-Play 
Assessment 


FOR  OFFICIAL  USE  ONLY  (when  filled  in* 


Tactical  Iraqi  Role-Play  Task  2:  Introducing  Your  Team  and  Building  Initial  Rapport 
Rater  Instructions 

Ask  the  interviewee  to  complete  Task  2  in  Iraqi  Arabic.  Please  respond  appropriately  in  the  role-play  as  a  "typical"  Iraqi  civilian  on 
the  street  would  respond.  At  the  conclusion  of  Task  1  mark  whether  or  not  the  interviewee  successfully  achieved  each  objective 
Then,  rale  the  interviewee's  speech  across  oil  objectives  on  the  scales  below.  Your  ratings  should  be  based  on  Task  2  only.  NOT 
the  OPI  results  or  results  on  other  tasks.  Your  rating  should  reflect  performance  within  the  narrow  context  and  content  of  Task  2. 
Please  consult  the  assessment  protocol  for  the  instructions  to  the  interviewee  for  the  role-play,  complete  task  objectives,  and  more 
detailed  instructions. 


Task  Objectives 

M  Objective  I:  Introduce  your  team  to  the  Iraqi,  including  your  ranks. .. 

■  Objective  2:  Describe  yourself,  including  your  nationality  and  the  branch  of  the  military  in  which  you  serve 
■I  Objective  3:  Ask  Iraqi  about  his/her  family. 

■  Objective  4:  Describe  your  family  to  the  Iraqi  when  appropriate,  including  your  marital  status  and  children... 


Completed? 

Yes  No 

Yes  No 

Yes  No 

Yes  No 


's  Task  2  speech  oneach  of  the  following  scales.  Remember,  rate  the  speech  within  the  narrow  context 
a  global  proficiency  rating.  See  rater's  protocal  for  the  full  scale.  Mark  only  one  response  per  scale. 

1.  lexical  Control 

■  Memorized  words  and  phrases  only  that  are  related  to  this  specific  task  in  this  contxt. 

■  Vety  limited  but  sufficient  to  cover  the  basic  courtesy  expressions,  introductions,  identification.- 

I  Sufficient  to  communicate  with  case  on  topics  relevant  for  this  task  in  this  context  and  content  domain. 

■  Broad  enough  for  effective  formal  and  informal  conversations  on  practical,  social,  and  professional  topics... 


2.  Structural  Control 

■  No  control.  Can  only  use  memorized  structures 

I  Structural  accuracy  is  random  or  severe  ly  limited.  A  Imost  every  utterance  has  errors  in  basic  structures... 

■  Minimally  cohesive  discourse.  Grammatical  structures  are  usually  not  vety  elaboraC  &  not  thoroughly  controlled;  frequent  errors— 

■  Effectively  combines  structure  and  vocabulary  to  convey  meaning.  Discourse  is  cohesive... 

3.  Sociolinguistic  Competence 

■  Severely  limied.  Any  knowbdgc  of  cultural  appropriateness  has  a  nonlinguistic  source. 

I  Uses  greetings  and  courtesy  expressions.  Can  interact  with  native  speakers  used  to  dealing  with  non-natives. 

■  Satisfies  routine  social  demands  and  limited  work  requirements  in  the  context  of  the  role-play  tasks.. 

■  Uses  cultural  rctcrcncev  When  errors  arc  made,  can  easily  repair  the  conversation. 

4.  Delivery 

I  Even  in  memorized  speech,  stress  intonation,  and  tone  usually  quite  f  aulty. 

B  Often  speaks  with  great  difficulty.  Pronunciatioa  stress,  intonation  generally  poor, 
a  Speaks  with  confidence  but  not  facility.  Can  usually  be  understood  by  those  not  used  to  dealing  » ith 
a  Speaks  readily  and  fills  pauses  suitably  .  Pronunciation  may  be  obviously  foreign... 

5.  Text  Produced 

B  Individual  words  and  phrases. 

B  Discree  sentences. 

B  Full  paragraphs. 

■  Extended  discourse. 


FOR  OFFICIAL  USE  ONLY  <»bcn filled  m, 


(Overt 


Cof^ right  C  Surface.  Want  S  Atsocum.  MtM-2007.  All  right,  reserved. 
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Choose  appropriate  location 


r  ' 

•  Need  for  control  vs.  authenticity  and  fidelity 

_ _ _ 
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Assessment  Design 


•  Promote  standardization 


step  3 


Task  I:  Greeting  an  Iraqi  Chilian 

1.  Read  in  English  the  instructions  and  performance  objectives  for  Task  1  in  English. 
READ  THIS  TO  THE  INTERVIEWEE: 

“For  Task  1.  you  are  an  American  Soldier  on  the  street  of  the  fictitious  Iraqi  town  of  Al- 
Wardiya.  You  arc  in  full  gear  and  wearing  sunglasses.  You  see  an  Iraqi  civilian  on  the 
street  and  you  approach.  I  will  be  the  Iraqi  civilian.  Your  goal  will  be  to  achieve  the 
following  objectives  to  the  best  of  your  ability  using  Iraqi  Arabic." 

Task  objectives  - 

•  “Objective  1:  Greet  the  Iraqi  using  a  formal  greeting" 

•  “Objective  2:  Introduce  yourself  by  slating  your  name" 

•  “Objective  3:  Ask  the  Iraqi  civilian  what  his  or  her  [tester  gender]  name  is" 

•  “Objective  4:  Respond  appropriately  to  what  the  Iraqi  civilian  says 
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Select  appropriate  activities 


•  Equal  opportunity  to  perform 


Must  provide  an  opportunity  to  demonstrate  the  desired  behavior 
and  to  meet  the  assessment  objective,  especially  if  Summative. 

But,  could  be  designed  to  be  observational  learning  opportunity.  Or, 
could  be  team  or  role-based  activity.  Must  fit  objectives. 


Ask  on  participants  (leaners,  role  players,  assessors)  if  each 
learner  had  the  opportunity  to  perform  or  completed  each  activity 
or  objective: 


Task  Objectives 

Completed 

•} 

• 

Objective  1:  Greet  the  Iraqi  using  a  formal  greeting. 

Yes 

No 

Objective  2:  Introduce  yourself  by  stating  your  name. 

Yes 

No 

Objective  3:  Ask  Iraqi  what  his  or  her  [tester  gender]  name  is. 

O  Yes 

No 

Objective  4:  Respond  appropriately  to  what  the  Iraqi  civilian  says. 

O  Yes 

No 
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•  Promote  standardization 

•  Proper  elicitation  of  targeted  language  skills 


•  Research  related  to  assessment  centers  in  the  I/O  field 
has  shown  that  a  common  challenge  is  for  role  players  to 
elicit  appropriate  behaviors.  Therefore,  training  is 
required  (Schollaert  &  Lievens,  2011). 

•  “Role-players  should  play  objectively  and  consistently  the 
role”  (International  Task  Force  on  Assessment  Center 
Guidelines,  2009,  p.  248). 

•  Role  players  allow  learners  to  demonstrate  the  desired 
performance  or  lack  thereof.  Assessors  consistently  and 


02.21.2014 


effectively  apply  a  ratingjubdp  i 


2014011201 


©  SWA  Consulting  Inc.  2014 
esurface@swa-consulting.com 


Slide  14 


Assessment  Design 
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Develop  appropriate  scales  and  evaluation  rubrics 


Consulting 


r 


•  Link  to  objectives 

•  Shared  mental  model  for  all  raters— training 

•  For  formative  assessments: 

•  Behaviorally-based 

•  Include  learner  in  the  process 

•  Could  be  competency-based 


Example  #1 : 


3.  Sociolinguistic  Competence 

Severely  limited.  Any  knowledge  of  cultural  appropriateness  has  a  nonlinguistic  source. 

Uses  greetings  and  courtesy  expressions.  Can  interact  with  native  speakers  used  to  dealing  with  non-natives. 
Satisfies  routine  social  demands  and  limited  work  requirements  in  the  context  of  the  role-play  tasks... 

Uses  cultural  references.  When  errors  are  made,  can  easily  repair  the  conversation. 
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From  instructor  perspective... 


Conducting  simple  conversations  to  build  rapport. 
Giving  commands . 


Did  not  meet 
expectations 

Approached 

expectations 

Met 

expectations 

Exceeded 

expectations 

N/A 

..  o 

o 

o 

o 

o 

••  o 

o 

o 

o 

o 

Use  your  notes  about  the  student’s  quality  of  language  use  to  provide  detailed  feedback  that  the  student  can  use  to  further 
develop  his  language  skills. 

Please  list  the  student's  top  3  effective  behaviors.  Then,  explain  why  each  behavior  was  effective. 


From  student  perspective... 


Not 

Slightly  Moderately 

Prepared 

Very 

N/A 

Prepared 

Prepared 

Prepared 

Prepared 

Conducting  simple  conversations  to  build  rapport . 

.  o 

o 

o 

o 

o 

o 

Giving  commands . 

.  o 

o 

o 

o 

o 

o 
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•  Use  of  technology  for  capturing  assessment 
performance  and  rating... 

-  Audio 

-  Video  (e.g.,  Go  Pro) 

•  Feedback  and  score  reports  must  fit  purpose 

•  Learner  Perceptions 

—  Relevant 
—  Fairness 

-  Accuracy 
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